Semantic-SAM Repository Application

My repository: https://github.com/Chen-Yulin/Semantic-SAM
My venv: ssam

Installation

测试过的python版本：3.8,3.10
官方步骤：

pip3 install torch==1.13.1 torchvision==0.14.1 --extra-index-url https://download.pytorch.org/whl/cu113
python -m pip install 'git+https://github.com/MaureenZOU/detectron2-xyz.git'
pip install git+https://github.com/cocodataset/panopticapi.git
git clone https://github.com/UX-Decoder/Semantic-SAM
cd Semantic-SAM
python -m pip install -r requirements.txt

export DATASET=/pth/to/dataset  # path to your coco data

一些绊脚石 ^ ^

1

根据[[Cuda+Torch]]，需要先安装cudatoolkit和cuda-toolkit

conda install nvidia/label/cuda-11.7.0::cuda-toolkit -c nvidia/label/cuda-11.7.0 
conda install cudatoolkit # no need to specify version
conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/<name>/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/<name>/include/" # `/usr/include`for missing `crypt.h`
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/<name>/"

然后按照torch官网的安装指令：

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

2

第二行直接运行可能会报错，提示系统gcc版本过高，安装gcc=11.2.0

conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

# 指定编译器路径
export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

# 找不到crypt.h的情况
sudo pacman -S libxcrypt-compat

export CXXFLAGS="${CXXFLAGS} -fuse-ld=/usr/bin/ld"

如果编译时出现ld: cannot find -lcudart: No such file or directory collect2: error: ld returned 1 exit status 报错，只是因为没有安装cudatoolkit ^ ^

3

安装完成后直接import semantic_sam会报错ModuleNotFoundError: No module named 'MultiScaleDeformableAttention' ^ ^
提示：

Please compile MultiScaleDeformableAttention CUDA op with the following commands:
	`cd mask2former[/modeling/pixel_decoder/ops](http://127.0.0.1:8888/modeling/pixel_decoder/ops)`
	`sh make.sh`

需要手动make一下 Mask2Former:

cd Mask2Former/mask2former/modeling/pixel_decoder/ops/
sh make.sh

4

一些版本问题

pip install gradio==3.37.0
pip install matplotlib==3.7.0

Demo 🐱

Generate multi-granularity Mask on CLICK

python demo.py --ckpt ./weights/swinl_only_sam_many2many.pth

Comment: 效果相较于SAM更多体现了语义的一致性，而不是基于texture进行分割。

Automatically Generate Mask on Different Granularity

python demo_auto_generation.py --ckpt ./weights/swinl_only_sam_many2many.pth

需要解决的问题

同level下mask重合

Solved by `utils.psg_utils.mask.discard_submask` in `psg_data.segment_pipeline`

Part-seg-dataset Generation

效果

原图：

.png)

Instance identification:

Part segmentation